feat: model-quality campaign — per-stage model routing + closed-loop targeted repair#42
feat: model-quality campaign — per-stage model routing + closed-loop targeted repair#42miguelgfierro wants to merge 42 commits into
Conversation
…ccept guard, never erase to null, compact array evidence
feat: per-stage model routing for the extraction pipeline
feat: closed-loop targeted repair of judge/validator-failing fields
Review — mechanism is solid, the challenge is on the default policy and repair costBoth features are well designed and unusually well tested (the edge cases that normally slip through here — null-candidate never erasing, judge-omitted candidate staying UNCERTAIN, compact array evidence, validator-only path — are all covered). The direction is right. My challenge is not on the code but on (a) the I verified the funnel claim directly: Feature 1 — per-stage model routingClean implementation, and The shipped
Concrete numbers from our last PROD window (Jul 7-8, 1594 pages): main pipeline was $28.32 all-sonnet (judge ~$11.65) and Suggested tiering: Also worth documenting for backend consumers: "pinned wins over Feature 2 — closed-loop targeted repairRight technique, and the monotonic accept guard is careful work. Three things to address before a broad rollout:
Minor: Suggested sequencingShip routing first (low risk, immediate observability/cost benefit) with the corrected tiering, then repair (opt-in, larger surface) with page-slicing and a scope threshold. Neither change touches the PR's architecture. |
Flagged-but-PASS fields are often ambiguous-but-correct; repairing them is spend the accept guard usually discards. FLYDOCS_REPAIR_INCLUDE_FLAGGED (default true, preserving current behaviour) lets cost-sensitive deployments limit repair to judge FAIL + validator errors.
Without a cap, an all-fail task degenerates into a full re-extract plus a full judge re-check on the repair model, with escalation still able to fire afterwards -- strictly costlier than judge->escalation alone. Above the cap (default 0.5) repair now steps aside for that task and leaves the FAIL verdicts for judge_escalation; RepairInfo.tasks_skipped records it.
Repair fanned out one unbounded gather over all failing tasks; FLYDOCS_REPAIR_TASK_CONCURRENCY (default 4) now caps in-flight repair passes, mirroring bbox_refine_doc_concurrency for provider rate limits.
Arrays are re-extracted and accepted whole, so a previously-correct row can be replaced by a new verifier-passing value -- monotonicity is field-level only. rows_changed records how many rows differ per repaired array so table repairs can be diffed in QA; the module docstring no longer overstates the guarantee.
The focused repair trimmed the schema subset but re-sent every page of the segment slice -- and pages, not fields, dominate multimodal input cost. When every failing field carries page provenance and the media is PDF, the repair (extract + judge re-check) now runs on a sub-slice spanning the failing pages plus one page of margin, and accepted candidates' slice-relative pages are remapped to segment coordinates. Falls back to the full slice for non-PDF media, unknown pages (nulled values lose theirs), non-shrinking spans, or slicing errors.
The repair pass eats into the hard FLYDOCS_SYNC_TIMEOUT_S wall (the pipeline is cancelled at the ceiling with a deterministic 408), so enabling stages.repair on the sync channel raises 408 likelihood. RequestValidator.validate() gains a sync flag, set by the sync controller and the /extract:validate dry-run; the async submit path is unchanged.
…v_template Splitter, classifier and the bbox value matcher are mechanical work and run fine on haiku; extraction/reasoning stay on sonnet and the judge on opus, so the per-stage lever cuts cost instead of only raising it. Also state that a pinned stage wins over options.model, correct the repair predicate wording (flag_for_review included), and document the three new repair settings plus the sync-cap note.
…cing notes payload-reference and deployment now state that a pinned FLYDOCS_<STAGE>_MODEL wins over the request's options.model (previously only a code comment). pipeline.md documents the repair page sub-slice, the failing-fraction scope cap, the concurrency bound, the include-flagged knob and the sync-latency warning.
|
Thanks — every point actioned in #43 (feat/pr42-review-fixes → demo), which will flow into this PR once merged. No architecture changes. Routing: Repair: (2.1) page-slices to the failing fields' pages via 336 unit tests green, ruff clean. |
Integration PR for the
demobranch: two composable features that raise extraction quality by putting the right model on the right task and closing the verification loop. Verified together on this branch (325 unit tests green, incl. the cross-feature orchestrator harness).1. Per-stage model routing — #40
What it does. Every LLM pipeline stage (splitter, classifier, extract, visual/content authenticity, judge, rules, transform, bbox matcher) can be pinned to its own model via a
FLYDOCS_<STAGE>_MODELenv var. Any stage left unset falls back to the request'soptions.model, then the globalFLYDOCS_MODEL. A pinned stage wins overoptions.model, so operator stage-tuning survives per-request overrides.env_templateships a recommended tiering: sonnet as the floor for every stage,anthropic:claude-opus-4-8for the judge.Benefit. Cost and quality stop being one global trade-off. Mechanical stages no longer have to pay for the strongest model, while verification — where a stronger, different model catches errors the extractor is systematically blind to — gets exactly that.
pipeline.usage.by_modelalready splits cost per model id, so the spend per tier is visible on every response with no extra work.Details: see the full description in #40.
2. Closed-loop targeted repair — #41
What it does. A new opt-in
repairstage (options.stages.repair) betweenjudgeandjudge_escalation. Fields that failed the judge re-check or a deterministic validator are re-extracted in ONE focused pass whose prompt quotes each previous value and the concrete reason it was rejected (judge evidence + validator errors). Repaired candidates are re-verified, and replace the originals only under a monotonic accept guard: validators pass, an explicit judge PASS when the judge stage is on, and never a null value. The pass runs on its ownFLYDOCS_REPAIR_MODEL(opus recommended) and reports apipeline.repairaudit block.Benefit. The pipeline already detects bad values twice (judge verdicts with written evidence, validator errors) but previously never acted on that evidence field by field — the only recovery was
judge_escalation, a blind whole-document re-run that re-rolls the dice on fields that were right. Targeted repair fixes the specific mistakes at a fraction of that cost, can only improve the result (never degrade it), and because it runs before escalation, the expensive full re-run only fires when targeted repair was not enough.Details: see the full description (including the adversarial-review findings and fixes) in #41.
How they compose
The intended production policy becomes: default model extracts → strong model judges → strong model repairs only what failed — the quality of a second full pass at a fraction of its cost, with a full audit trail (
pipeline.repair,pipeline.escalation,pipeline.usage.by_model).One reconciliation commit on this branch adapts #40's orchestrator test harness to the
field_repairerconstructor parameter introduced by #41.